The present invention pertains to neural networks. In particular, the invention pertains to neural networks in complex data processing requiring associative recall, supervised learning, or optimization. More particularly the invention pertains to directing such processing operations through masking by imposing weighting on individual neurons, that is, affecting their thresholds.
A neural network consists of very simple processing elements (neurons). FIG. 1a shows a biological neuron 10. The basic electronic neuron and circuitry 20 is illustrated in FIG. 1b. The neurons are commonly interconnected in a network topology 30, as shown in FIG. 2, so that each neuron receives inputs from (many) other neurons. The points of interconnection (synapses) 12 are individually weighted. The weights and procedures used to adjust the weights determine the operation of the network 30. Each neuron 14 consists of a summing function and a threshold function. The summation function adds the values present on the neuron's inputs. The threshold function is usually a sigmoid transfer function (ramped step) 16, as shown in FIG. 3 so that when the summation function output exceeds the threshold, the neuron "fires" and its output switches from an off state to an on state. The potential power of neural networks lies in the parallel operation of a large number of neurons which operate collectively to solve an otherwise intractable problem.
Most neural networks are simulated in software rather than actually implemented. Since this simulation occurs on conventional computers, it can be quite inefficient. Emulation engines are available for several computer systems. These engines are usually conventional processors which have architectures tuned to the specific computations which must be performed during the simulation of neural networks. Notable exceptions are recently developed integrated circuits implemented at Bell Laboratories, California Institute of Technology, and the Jet Propulsion Laboratory.
Typical applications fall into three categories: recall, learning, and optimization. Networks in the recall category perform a mapping from one set of values to another (or the same) set of values. One example is an associative memory network popularized by J. J. Hopfield. These networks associate an initial neural state with a final neural state. The synaptic weight values define the associative mappings Hopfield showed that the operation of the network can be described by an energy function similar to gravity or an electromagnetic field, but in an N-dimensional space where N is the number of neurons. In Hopfield's network there is one field of neurons, the synaptic weights are symmetrical, and the neurons are updated asynchronously. A similar network, the Bi-dimensional Associative Memory (BAM), by B. Kosko, uses two interconnected fields on neurons, one for input and another for output. This configuration allows asymmetrical synaptic weights and parallel neural updating.
Learning networks perform a similar mapping function, but the weight values are initially small random values. A learning algorithm is used to adaptively adjust the weights as training stimuli ar applied. The algorithm compares the network's output with the desired response and determines the appropriate synaptic weight adjustment. An example of this type of network is the Boltzmann Machine. It contains three layered fields of neurons: input, hidden, and output. The synaptic connections are directed from input, through the hidden layer, to the output. No feedback connections are normally present in the Boltzmann Machine, through there have been studies of more general learning networks with feedback One popular learning algorithm, called Back Propagation, proceeds from errors at the output and iteratively adjust weights in the direction of the input. This results in the neurons in the hidden layer becoming sensitized to "features" of the problem as evidenced in the training stimuli.
Optimization networks usually have specialized topologies and weight relationships that represent the constraints of a particular problem. The network is started with its neurons in random states and allowed to operate until it settles on a final state that indicates an optimal solution to the problem. Hopfield describes a network which solved the travelling salesman problem (route planning) effectively. It did not necessarily find the optimal solution, but normally found "good" solutions very rapidly.
Control of a neural network is normally applied by its design or by a set of synaptic weight values. Once the net operation is started it proceeds to its conclusion with little external intervention. In simulation, initial data values are merely applied programmatically to variables representing the neuron outputs. This effectively sets the neurons to desired states. In physical systems, additional neural inputs are provided. Values are held on these inputs until the network stabilizes. The inputs are then removed and the net undergoes state transitions as determined by the neural states and the synaptic weights.